feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK by yahya-mouman · Pull Request #12318 · DataDog/dd-trace-java

yahya-mouman · 2026-08-27T14:01:13Z

Summary

Adds annotateAgentManifest(LLMObs.AgentManifest) to the Java LLMObs SDK, allowing users to manually declare an agent span's configuration (name, instructions, model, model_settings, tools). Mirrors the Python implementation (DataDog/dd-trace-py#19771) and follows the annotatePrompt pattern from #12161.

Changes

LLMObsTags.java — adds AGENT_MANIFEST = "agent_manifest" constant
LLMObs.java — adds two public immutable builder classes:
- LLMObs.AgentTool — represents a single tool (name, optional description, optional parameters)
- LLMObs.AgentManifest — builder with name, instructions, model, model_settings, tools
LLMObsSpan.java — adds default void annotateAgentManifest(LLMObs.AgentManifest) (no-op default for backwards compat)
NoOpLLMObsSpan.java — explicit @Override no-op
DDLLMObsSpan.java — real implementation: validates span kind (agent only), builds manifest map, stores as _ml_obs_tag.agent_manifest
LLMObsSpanMapper.java — adds agent_manifest to TAGS_FOR_REMAPPING; serializes to meta.agent_manifest as a msgpack map

Behaviour

LLMObs.AgentManifest manifest = LLMObs.AgentManifest.builder()
    .name("travel_desk")
    .instructions("Book travel for the user.")
    .model("gpt-4o")
    .modelSettings(Map.of("temperature", 0.7))
    .tools(List.of(LLMObs.AgentTool.from("get_weather", "Look up weather", null)))
    .build();

agentSpan.annotateAgentManifest(manifest);

Only applies to agent span kind; other span kinds emit a log warning and no-op
Calling twice on the same span overwrites the previous manifest (no merge)
framework is set to "AgentObs SDK" automatically by the SDK
name defaults to the span name if not provided
model_settings keys are forwarded as-is (no allowlist in this initial PR)
Null manifest is silently ignored

Test plan

./gradlew :dd-trace-api:test --tests "datadog.trace.api.llmobs.LLMObsTest" — 27 tests pass (4 new)
./gradlew :dd-java-agent:agent-llmobs:test --tests "datadog.trace.llmobs.domain.DDLLMObsSpanTest" — 39 tests pass (7 new)
./gradlew :dd-trace-core:test --tests "datadog.trace.llmobs.writer.ddintake.LLMObsSpanMapperTest" — 18 tests pass (2 new)

🤖 Generated with Claude Code

- Add AgentManifest immutable value class with Builder pattern in LLMObs.java - Add AgentTool immutable value class in LLMObs.java - Add AGENT_MANIFEST constant to LLMObsTags - Add annotateAgentManifest() default method to LLMObsSpan interface - Add comprehensive builder tests for AgentManifest and AgentTool Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Stores agent manifest fields (name, instructions, model, model_settings, tools) as an internal tag `_ml_obs_tag.agent_manifest`. Only applies to agent spans; warns and no-ops on other span kinds. Null manifest is silently ignored. Tools with null/empty names are skipped with a warning. A second call overwrites the previous manifest. Framework field "AgentObs SDK" is always added when any manifest fields are present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add AGENT_MANIFEST_KEY byte constant, include the tag in TAGS_FOR_REMAPPING, and handle it in the meta serialization loop as a msgpack map. Add two tests: one verifying all manifest fields appear in meta, one verifying the tag does not leak into the tags list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

datadog-prod-us1-6 · 2026-08-27T14:16:43Z

🎯 Code Coverage (details)
• Patch Coverage: 82.42%
• Overall Coverage: 58.36% (-0.42%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: b5f56f1 | Docs | View more details | Give us feedback!}

ncybul · 2026-08-27T16:51:14Z

@codex

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5f56f11d6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-27T16:54:54Z

+    String name =
+        manifest.getName() != null ? manifest.getName() : (sn != null ? sn.toString() : null);
+    if (name != null && !name.isEmpty()) {


Fall back to the span name for empty manifest names

When a caller supplies .name(""), such as by forwarding an optional configuration value, this selects the empty string instead of the span-name fallback, and the following guard then omits name entirely. The serialized agent manifest is therefore nameless despite the API's default-to-span-name behavior; treat both null and empty names as absent before selecting the fallback.

Useful? React with 👍 / 👎.

ncybul

A couple minor suggestions but overall looks good to me! Would be nice to see a manual verification of setting the manifest via the SDK.

ncybul · 2026-08-27T17:51:51Z

+   *
+   * <p>Build via {@link AgentManifest#builder()} and pass to {@link
+   * LLMObsSpan#annotateAgentManifest(AgentManifest)}. Only applied on agent spans; ignored on other
+   * span kinds. A subsequent call on the same span overwrites the previous manifest.


This is different from the Python implementation, right? I think in Python, we were merging the two manifests together whenever possible. It's probably best that we align the implementations and choose one approach.

Personally, I would lean towards merging the fields whenever possible.

Yes we can do that. It'll help the async distributed trace cases too,

ncybul · 2026-08-27T17:53:15Z

+    String name =
+        manifest.getName() != null ? manifest.getName() : (sn != null ? sn.toString() : null);


What happens when name is an empty string here?

If name is supplied we use it. I guess if its an empty string we should still treat that as a null and use the span name.

ncybul · 2026-08-27T17:55:25Z

  private static final String METADATA = LLMOBS_TAG_PREFIX + LLMObsTags.METADATA;
  private static final String TOOL_DEFINITIONS = LLMOBS_TAG_PREFIX + LLMObsTags.TOOL_DEFINITIONS;
+  private static final String AGENT_MANIFEST = LLMOBS_TAG_PREFIX + LLMObsTags.AGENT_MANIFEST;
+  private static final String MANUAL_FRAMEWORK = "AgentObs SDK";


Idk if "AgentObs SDK" makes sense here or if we should default to "custom" or "manual" since the former makes it seem like we are determining the agent manifest via the SDK rather than the user supplying it. (same comment applies to the python PR)

Yeah I had a similar thought about this thanks for confirming !

dd-octo-sts · 2026-08-28T11:08:53Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

datadog-prod-us1-6

More details

The default API method keeps compatibility. The implementation accepts manifests only for agent spans and writes them to meta.agent_manifest.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit b5f56f1 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

yahya-mouman and others added 4 commits August 27, 2026 13:36

feat(llmobs): add annotateAgentManifest default method to LLMObsSpan

ff0f77c

yahya-mouman added the type: feature Enhancements and improvements label Aug 27, 2026

yahya-mouman requested review from ncybul and sabrenner August 27, 2026 14:15

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

ncybul approved these changes Aug 27, 2026

View reviewed changes

yahya-mouman marked this pull request as ready for review August 28, 2026 11:08

yahya-mouman requested a review from a team as a code owner August 28, 2026 11:08

dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Aug 28, 2026

datadog-prod-us1-6 Bot reviewed Aug 28, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK - #12318

feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK#12318
yahya-mouman wants to merge 4 commits into
masterfrom
yahya/llmobs-agent-manifest-manual

yahya-mouman commented Aug 27, 2026

Uh oh!

datadog-prod-us1-6 Bot commented Aug 27, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

ncybul commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Uh oh!

ncybul left a comment

Uh oh!

ncybul Aug 27, 2026

Uh oh!

yahya-mouman Aug 28, 2026

Uh oh!

ncybul Aug 27, 2026

Uh oh!

yahya-mouman Aug 28, 2026

Uh oh!

ncybul Aug 27, 2026

Uh oh!

yahya-mouman Aug 28, 2026

Uh oh!

dd-octo-sts Bot commented Aug 28, 2026

Uh oh!

datadog-prod-us1-6 Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		String name =
		manifest.getName() != null ? manifest.getName() : (sn != null ? sn.toString() : null);

Uh oh!

Conversation

yahya-mouman commented Aug 27, 2026

Summary

Changes

Behaviour

Test plan

Uh oh!

datadog-prod-us1-6 Bot commented Aug 27, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncybul commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

ncybul left a comment

Choose a reason for hiding this comment

Uh oh!

ncybul Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

yahya-mouman Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

ncybul Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

yahya-mouman Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

ncybul Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

yahya-mouman Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Aug 28, 2026

Uh oh!

datadog-prod-us1-6 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-prod-us1-6 Bot commented Aug 27, 2026 •

edited by datadog-prod-us1-4 Bot

Loading